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Literacy skills are critical to students’ academic achievement 
and setting them on a path to successful high school 
graduation and readiness for college and careers. Web- 

Based Intelligent Tutoring for the Structure Strategy (ITSS) is a 
supplemental web-based program for students in grades Ic8. 

It is designed to develop literacy skills needed to understand 
factual texts encountered in classrooms and everyday life. The 
program teaches students how to follow the logical structure of 
factual text and to use text structure to improve understanding 
and recall. In particular, ITSS highlights five main text 
structures that are used to (1) make comparisons; (2) present 
problems and solutions; (3) link causes and effects; (4) present 


sequences; and (5) describe things, people, creatures, places, 
or events. The program helps students classify the structure of 
a passage by identifying certain key words, such as “solution” 
and “in contrast,” that clue readers in to the type of arguments 
the text is making. 


This What Works Clearinghouse (WWC) report, part of the 
WWC’s Adolescent Literacy topic area, explores the effects of 
ITSS on comprehension. The WWC identified two studies of 
ITSS that meet WWC standards.’ The evidence presented in 
this report is from studies of the impacts of ITSS on students 
in grades 4, 5, and 7 and a variety of school settings, including 
suburban and rural districts.* 


What Happens When Students Participate in ITSS?* 


The evidence indicates that implementing ITSS is likely to 
increase students’ comprehension. 


Findings on ITSS from two studies that meet WWC standards 
are shown in Table 1. The table reports an effectiveness 
rating, the improvement index, and the number of studies 


and students that contributed to the findings. The improve- 
ment index is a measure of the intervention’s effect on an 
outcome. It can be interpreted as the expected change in 
percentile rank for an average comparison group student if 
that student had received the intervention. 


Table 1. Summary of findings on ITSS from studies that meet WWC Standards 


Study findings 


Evidence meeting WWC standards (version 4.0) 


Outcome domain Effectiveness rating 


Improvement index 
(percentile points) 


Number of studies Number of students 


Comprehension Positive effects 


+9 2 6,724 


Note: The improvement index can be interpreted as the expected change in percentile rank for an average student in the comparison group if that student had received the 
intervention. For example, an improvement index of +9 means that the expected percentile rank of the average comparison group student would increase by 9 points if the 
student received /TSS. The improvement index values are generated by averaging findings from the outcome analyses that meet WWC standards, as reported by Wijekumar, 
Meyer, and Lei (2012, 2017). A positive improvement index does not necessarily mean the estimated effect is statistically significant. Comprehension outcomes include the Gray 
Silent Reading Test and 13 researcher-designed measures that test students’ ability to understand written text by identifying main ideas, problems and solutions, and assessing 
the use of structure to organize correct ideas. The effects of /TSS are not known for other outcomes within the adolescent literacy topic area, including alphabetics, reading 
fluency, general literacy achievement, writing conventions, writing productivity, and writing quality. 


BOX 1. HOW THE WWC REVIEWS AND DESCRIBES EVIDENCE 


The WWC evaluates evidence based on the quality and results of reviewed studies. The criteria the WWC uses for evaluating 
evidence are defined in the Procedures and Standards Handbooks and the Review Protocols. The studies summarized in this report 
were reviewed under WWC Standards (version 4.0) and the Adolescent Literacy topic area protocol (version 4.0). 


To determine the effectiveness rating, the WWC considers what methods each study used, the direction of the effects, and the 
number of studies that tested the intervention. The higher the effectiveness rating, the more certain the WWC is about the reported 
results and about what will happen if the same intervention is implemented again. The following key explains the relationship between 
effectiveness ratings and the statements used in this report: 


Effectiveness rating Rating interpretation 


Description of the evidence 


Positive (or negative) effects The intervention is likely to change an Strong evidence of a positive effect, with no 


outcome 


overriding contrary evidence 


Potentially positive (or negative) effects The intervention may change an outcome Evidence of a positive effect with no overriding 


contrary evidence 


No discernible effects The intervention may result in little to no No affirmative evidence of effects 


change in an outcome 


Mixed effects The intervention has inconsistent effects Evidence includes studies in at least two of 


onan outcome 


How is ITSS Implemented? 


The following section provides details of how ITSS was 
implemented. This information can help educators identify 
the requirements for implementing ITSS and determine 
whether implementing this intervention would be feasible 
in their districts or schools. Information on ITSS presented 
in this section comes from studies that meet WWC group 
design standards (Wijekumar et al., 2012, 2017) and from 
correspondence with the developer. 


* Goal: ITSS aims to improve comprehension of factual 
(nonfiction) text. 


¢ Target population: ITSS is designed for students in 
grades K-8. The two studies that contribute to this inter- 
vention report included students in grades 4, 5, and 7. 


¢ Method of delivery: ITSS is a web-based program, which 
students access on a personal computer with earphones. 
It is typically used during class to supplement the English 
language arts curriculum. 


these categories: studies with positive effects, 
studies with negative effects, or more studies 
with indeterminate effects than with positive or 
negative effects 


Comparison group: In the two studies that con- 
tribute to this intervention report, students in the 
comparison group received the school’s standard 


language arts curriculum. Additional information on 
these curricula is not available. Total daily and weekly 
amounts of language arts instruction were the same 
for both intervention and comparison classrooms. 


Frequency and duration of service: Developers 
recommend students use ITSS twice a week for about 30 
minutes per session for 7 months to 1 year. Refer to Table 
2 for additional details. 


Intervention components: ITSS includes several compo- 
nents (see Table 2). 


Table 2. Components of ITSS 


Key component 


Description 


Instruction 


Exercises with 
immediate feedback 


ITSS models how to use the structure of the factual text to improve understanding and recall. An animated person, called 

an Intelligent Tutor (I.T.), explains the five main text structures: comparisons, problems and solutions, cause and effect, 
sequences, and descriptions. Then, the |.T. displays and reads aloud a passage that illustrates a text structure, modeling 
appropriate techniques for understanding the displayed passage. Techniques include finding signaling words, identifying the 
text structure, understanding the main idea of the text, and summarizing the text in written form. /TSS provides approximately 
12 lessons for each text structure. 


ITSS offers practice exercises using passages from a variety of substantive areas, including science, social studies, sports, 
and current events. It assesses student progress and provides learners with immediate feedback. After completing a lesson, 
the program allows students to practice what they learned in a series of exercises, and students work at their own pace. Each 
exercise involves reading a passage and then completing a series of tasks, such as finding signaling words, describing the 
main idea of the text, and summarizing the passage. 


Most exercises in /TSS follow this sequence: 


1. Students read nonfiction text displayed on the screen. 
2. Students identify the author's top-level text structure (i.e., comparison, problem and solution, cause and effect, 


sequence, or description). 


3. Students select signaling words used in the text. 


4. Students write a sentence summarizing the main idea of the text, which is displayed on the screen, using the 


structure strategy. 


5. Students generate a thorough summary of the text using the /TSS comprehension tools, such as an annotated matrix, 
diagram, sequence texts, or texts with embedded text structures. 


6. Students write a “recall” summary of the text after the text is removed from view. This recall exercise becomes more 
challenging as students progress through the lessons. In early lessons, students recall and summarize the text while 
having access to their (or the I.T.’s) main idea summaries from steps 4 or 5. Later practice lessons remove this aid, and 
students monitor their recall using the structure strategy. More specifically, students use the text structure as a retrieval 
and writing guide. They monitor their understanding and recall through summarizing the main points according to the 
recall pattern identified for a particular text structure. 


7. Students receive feedback from the I.T. after each step above. In the two studies that contributed to this report, after 
several unsuccessful attempts, the |.T. offered students hints, showed a model summary of the text, and asked students 
to correct their work. The students were not allowed to copy the model summary but were asked to think about the 
text’s structure and main idea and then revise their work. If students were unsuccessful after repeated feedback and 
increased help from the |.T., the I.T. told students to ask teachers for assistance. 

The tasks can vary across lessons. Some lessons ask students to write or select a good title for a text based on text structure, 


write their own texts for each text structure by selecting signaling words from a specified short list, and correct a fictitious 
student's muddled use of the text structure strategy. As students complete the exercises, the passages become more difficult. 


What Does ITSS Cost? 


This preliminary list of costs is not designed to be 


exhaustive; rather, it provides educators an overview of the 


major resources needed to implement ITSS. The program 
costs described below are based on the information 
available as of June 2019. 


¢ Personnel costs: ITSS developers require that school 
staff who would like to implement the ITSS program 
participate in professional development. In particular, 
teachers are required to attend a 2-day session, while 
school administrators are required to attend a 1-day 
session. The cost of professional development sessions 
varies from $2,500 to $8,000 per session depending 
on the number of school staff attending the session and 
the location of the sessions. An additional 4 to 6 days of 
in-school coaching is also required. No cost information 
on in-school coaching is available. 


Facilities costs: No additional facilities costs are needed 
to run the ITSS program; however, computer labs might 
be necessary for classrooms without personal computers. 


Equipment and materials costs: The ITSS web-based 
program can be accessed by students and educators at 
no cost. 


Costs paid by students or parents: Students and their 
parents do not pay to access ITSS. 


In-kind supports: No information on in-kind supports 
is available. 


Sources of funding: School districts typically cover ITSS 
training costs. 


For More Information: 
About [TSS 


Dr. Kausalai (Kay) Wijekumar, Texas A&M University, College of Education and Human Development, College Station, TX 77843 


Web: http://literacy.io/projects/itss/ 


About the cost of the intervention 
Cost information was provided by the developer. 


Research Summary 


The WWC identified two studies that investigated the 
effectiveness of ITSS (Figure 1): 


¢ 1study meets WWC group design standards without 
reservations 


¢ 1study meets WWC group design standards with 
reservations 


The WWC reviews findings on the intervention’s effects on 
eligible outcome domains from studies that meet standards, 
either with or without reservations. Based on this review, the 
WWC generates an effectiveness rating, which summarizes 
how the intervention impacts, or changes, a particular 
outcome domain. The WWC reports additional supplemental 


Figure 1. Effectiveness ratings for ITSS 
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Contribute to effectiveness ratings 


ITSS has positive effects on comprehension 


findings on the WWC website (https://whatworks.ed.gov), 
such as those the study authors reported separately for male 
and female students. These supplemental findings do not 
contribute to the effectiveness ratings. 


The two studies of ITSS that meet WWC group design 
standards reported findings on comprehension outcomes. 
No other findings in the studies met WWC group design 
standards within any outcome domain included in the 
Adolescent Literacy topic area.° Citations for the two studies 
reviewed for this report are listed in the References section, 
which begins on page 11. 


0 studies do not 0 studies are 


meet WWC 
standards 


Taksii(ehle) (amie 
review 


Do not contribute to effectiveness ratings 


The WWC determined that one study that meets WWC group design standards without reservations and one study that 


meets WWC group design standards with reservations showed evidence of a positive and statistically significant effect 
of TSS on comprehension (Wijekumar et al., 2012, 2017). 


Main Findings 


Table 3 shows the findings from two ITSS studies that 

meet WWC group design standards. The table includes 
WWC calculations of the mean difference, effect size, and 
performance of the intervention group relative to the 
comparison group. Based on findings from two studies that 


meet WWC group design standards, the effectiveness rating 
for the comprehension domain is positive effects, indicating 
strong evidence of a positive effect with no overriding con- 
trary evidence. These findings are based on 6,724 students. 


Table 3. Findings from studies of ITSS by outcome domain 


Mean 
(standard deviation) WWC calculations 
Sample Intervention © Comparison Mean Effect § Improvement 

Measure (study) Study sample size group group difference size index p-value 
Gray Silent Reading Fourth- and fifth- 4,856 32.12 30.42 1.70 0.15 +6 <.01 
Test (GSRT) grade students (11.52) (11.37) 
Short Comparison Fourth- and fifth- 4,040 4.25 3.71 0.54 0.23 +9 <.01 
Text: Competence Test grade students (2.39) (2.38) 
Short Comparison Fourth- and fifth- 4,035 3.46 2.65 0.81 0.51 +20 <.01 
Text: Main Idea Test grade students (1.58) (1.58) 
Short Comparison Fourth- and fifth- 4,073 26.17 22.37 3.80 0.23 +9 <.01 
Text: Total Recall Test grade students (16.98) (16.53) 
Problem/Solution Text: — Fourth- and fifth- 4,084 3.54 3.21 0.33 0.14 +6 <.01 
Competence Test grade students (2.36) (2.35) 
Problem/Solution Fourth- and fifth- 4,090 17.93 15.76 2.17 0.19 +7 <.01 
Text: Total Recall Test grade students (11.66) (11.60) 
Outcome average for fourth- and fifth-grade comprehension (Wijekumar et al., 2012)? 0.24 +9 ean 
Gray Silent Reading Seventh-grade 1,868 nr nr 2.12 0.17 +7 <.01 
Test (GSRT) students (11.57) (13.41) 
Long Comparison Seventh-grade 1,717 nr nr 0.31 0.19 +8 <.01 
Text: Competence Test students (1.68) (1.51) 
Long Comparison Seventh-grade 1,717 nr nr 0.17 0.19 +8 <.01 
Text: Number of students (0.98) (0.75) 
Issues Test 
Short Comparison Seventh-grade 1,718 nr nr 0.36 0.16 +6 <.05 
Text: Competence Test students (2.25) (2.24) 
Short Comparison Seventh-grade 1,718 nr nr 0.29 0.21 +8 <.05 
Text: Number of students (1.49) (1.28) 
Issues Test 
Short Comparison Seventh-grade 1,721 nr nr 0.39 0.28 +11 <.01 
Text: Main Idea students (1.42) (1.33) 
Competence Test 
Short Comparison Seventh-grade 1,721 nr nr 0.51 0.42 +16 <.01 
Text: Main Idea Test, students (1.50) (0.58) 
Number of Issues 
Problem/Solution Text: | Seventh-grade 1,716 nr nr 0.52 0.20 +8 <.01 
Competence Test students (2.58) (2.51) 
Outcome average for seventh-grade comprehension (Wijekumar et al., 2017)’ 0.23 +9 Statistically 

significant 
Outcome average for comprehension across all studies 0.23 +9 


Notes: For mean difference and effect size values reported in the table, a positive number favors the intervention group and a negative number favors the comparison group. 
The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are given the 
intervention (measured in standard deviations of the outcome measure). An indicator of the effect of the intervention, the improvement index can be interpreted as the expected 
change in percentile rank for an average comparison group student if that student had received the intervention. For example, an improvement index of +9 means that the 
expected percentile rank of the average comparison group student would increase by 9 points if the student received /TSS. A positive improvement index does not necessarily 
mean the estimated effect is statistically significant. Some statistics might not sum as expected due to rounding. nr = Not reported. 
@ Wijekumar et al. (2012) did not require corrections for clustering or difference-in-differences adjustments. A correction for multiple comparisons was needed but did not affect 
whether any of the findings were found to be statistically significant. The authors provided analytic sample sizes for fourth-grade student samples in response to a question 
from the WWC to the authors. The WWC calculated the intervention group mean for each grade’s analytic sample by adding the adjusted mean difference (an HLM level-2 


coefficient) to the unadjusted posttest mean for the comparison group. Findings for the combined fourth- and fifth-grade analytic samples, including means, standard deviations, 
and p-values, were calculated by the WWC. For consistency, the WWC used pretest standard deviations from each grade to calculate effect sizes for the combined grade 4 
and 5 findings because student-level posttest standard deviations were not reported for the fourth-grade sample. This study is characterized as having a statistically significant 
positive effect on comprehension because the WWC-calculated average effect size for the study is positive and statistically significant. 


> Wijekumar et al. (2017) did not require corrections for clustering or difference-in-differences adjustments. The p-values presented here were reported in the original study. A 
correction for multiple comparisons was needed but did not affect whether any of the findings were found to be statistically significant. Intervention and comparison group standard 
deviations presented for seventh-grade students were provided in response to a question from the WWC to the authors. However, the authors did not report posttest means for 
the analytic sample used in the HLM analysis, which were used to estimate the mean difference between the two study groups; thus, the WWC did not report intervention and 
comparison group means in this table. This study is characterized as having a statistically significant positive effect on comprehension because the WWC-calculated average 
effect size for the study is positive and statistically significant. For more information, please refer to the WWC Procedures and Standards Handbook, version 4.0, page 22. 


In What Context Was ITSS Studied? 


The following section provides information on the setting 
of the two studies of ITSS that meet WWC standards, and 
a description of the participants in the research. This 


WHERE THE STUDY WAS CONDUCTED 


2 studies, 6,724 students in 2 states 


information can help educators understand the context in 
which the studies of ITSS were conducted, and determine 
whether the program might be suitable for their setting. 


suburban and rural school districts 


Grades PK K 3 (4 @ 


Grade 4,5, &7 


: @ 


Details of Each Study that Meets WWC Standards 


This section presents details for each study of ITSS that 
meets WWC standards. These details include the full study 
reference, findings description, findings summary, and 
description of study characteristics. A summary of domain 
findings for each study is presented below, followed by a 
description of the study characteristics. These study-level 
details include contextual information around the study 
setting, methods, sample, intervention group, comparison 
group, outcomes, and implementation details. For additional 
information, readers should refer to the original studies. 


Research details for Wijekumar et al. (2012) 


Wijekumar, K., Meyer, B., & Lei, P. (2012). Large-scale 
randomized controlled trial with 4th graders using 


intelligent tutoring of the structure strategy to improve 
nonfiction reading comprehension. Educational Technol- 
ogy Research & Development, 60(6), 987-1013. Retrieved 
from https://eric.ed.gov/?id=E]986753. 


Findings from Wijekumar et al. (2012) show evidence of a 
statistically significant positive effect of ITSS in the compre- 
hension domain (Table 4). These findings are based on six 
outcomes and 4,856 students.® The findings and research 
details summarized for this study come from four related 
citations, including the primary study listed above. See the 
References section, which begins on page 11, for a list of all 
related publications. 


Table 4. Summary of findings from Wijekumar et al. (2012) 


Meets WWC group design standards without reservations 


Study findings 
; Sample Average Improvement Statistically 
Outcome domain size effect size index significant 
Comprehension 4,856 students 0.24 +9 Yes 


Table 5. Description of study characteristics for Wijekumar et al. (2012) 


WWC evidence 
rating 


Setting 


Methods 


Study sample 


Intervention 
group 


Comparison 
group 


Outcomes and 
measurement 


Meets WWC Group Design Standards Without Reservations. This is a cluster randomized controlled trial with low 
attrition.” For more information on how the WWC assigns study ratings, please see the WWC Procedures and Standards 
Handbooks (version 4.0) and WWC Standards Briefs, available on the WWC website. 


The study included fourth- and fifth-grade classrooms from 45 rural and suburban elementary schools, 12 school districts, 
and two states in the United States. 


The study used a cluster randomized controlled trial design. A volunteer sample of 259 classrooms in participating 
schools were randomly assigned to either provide supplemental instruction using /7SS or to implement business-as- 
usual classroom instruction in the study year and to implement /TSS in the following academic year. Random assignment 
occurred within each grade level and school. If a school had an odd number of classrooms for a given grade level, it was 
matched to a similar school with an odd number of classrooms and random assignment then took place across these 
schools. Altogether, 130 classrooms were randomly assigned to the intervention group and 129 classrooms were randomly 
assigned to the comparison group. At baseline, the intervention group included 2,949 students and the comparison group 
included 2,848 students.® For all outcome measures, the sample loss after random assignment (attrition) was within the 
acceptable threshold for the review: at the classroom level, overall attrition ranged from 5.8% to 19.3%, and differential 
attrition ranged from 3.9% to 4.8%. Overall non-response at the student level ranged from 5.0% to 14.3%, and differential 
attrition ranged from 3.4% to 4.3%. 


The number of students in the analytic sample, including both fourth- and fifth-grade students, differs by outcome, ranging 
from 4,035 to 4,856 students. 


No demographic or sample characteristics were provided on the fourth-grade analytic sample; however, the authors stated 
that the intervention and comparison groups were balanced on student socioeconomic status, number of English learners, 
percentage of racial/ethnic minorities, and gender. The 45 schools in the fifth-grade sample are composed of 8% to 14% 
racial/ethnic minorities and 39% to 44% socioeconomically disadvantaged students. 


Students in the intervention group received the /TSS program over the course of the 2009-10 school year. [TSS sessions 
took place for 30 to 45 minutes a week over 6 to 7 months, which was lower than the developer-recommended dosage, 
as a partial substitute for the regular language arts curriculum. In fourth-grade classrooms, the recall task of [TSS was 
removed from the lesson sequence after the first month of instruction because students had difficulty typing. In fifth-grade 
classrooms, the recall task was implemented as intended. 


Students in comparison classrooms received the typical language arts curriculum, which was the same curriculum used by 
the intervention group classrooms within the same school except for the partial substitution of /TSS. Total daily and weekly 
amounts of language arts instruction were the same for both intervention and comparison classrooms. 


Study authors reported findings on six outcome measures that are eligible for review under the Adolescent Literacy 

topic area. All assessments were administered to both fourth- and fifth-grade students. For all measures, pretests were 
completed at the beginning of the school year and posttests were completed immediately after the end of the intervention 
implementation (in April or May 2010). 


All outcomes were reviewed in the comprehension domain and included a standardized test, the Gray Silent Reading 
Test (GSRT; Wiederholt & Blalock, 2000),°which contained 15 passages and posed five questions for each passage to 
test students’ comprehension, and five researcher-designed measures. For the researcher-designed measures described 
below, interrater agreement was within the acceptable threshold for the review and ranged from 86% to 99%: 


¢ Short Comparison Text: Competence Test. This assessment measured whether students recalled the problem, 
its cause, and its solution described in the original text. Students were presented with one passage at pretest and a 
different passage at posttest that compared two ideas or things. Each passage in this assessment had 128 words, 
15 sentences, and 98 idea units. Competency scores ranged from 1 to 8 and described a student’s use of the structure 
strategy to organize correct ideas based on the recall task. 


¢ Short Comparison Text: Main Idea Test. This assessment asked students to write a two-sentence main idea after 
reading the passage from the Short Comparison Text: Competence Test. The assessment measured students’ ability to 
understand and summarize the main ideas in the passage. This assessment was rated on a 6-point scale. 


¢ Short Comparison Text: Total Recall Test. This assessment measured students’ recall ability by asking them to write 
down as many ideas as they could remember after reading the passage from the Short Comparison Text: Competence Test. 


¢ Problem/Solution Text: Competence Test. This assessment measured students’ ability to recall the correct problem, 
its cause, and its solution after they read a passage using the problem/solution structure. Competency scores ranged 
from 1 to 8 and described a student’s use of the structure strategy to organize correct ideas. Students were presented 
with one passage at pretest and a different passage at posttest. Each passage had 98 words and 72 idea units. 


Outcomes and ¢ Problem/Solution Text: Total Recall Test. This assessment measured students’ recall ability by asking them to write 
measurement down as many ideas as they could remember from a passage. The passage was the same as the one used in the 
(continued) Problem/Solution Text: Competence Test. 


Supplemental findings were reported for male students, female students, students reading below grade level, and students 
reading above grade level. These supplemental findings were reported separately for fourth- and fifth-grade students 

and are available on the WWC website (https://whatworks.ed.gov). The supplemental findings do not factor into the 
intervention’s rating of effectiveness. 


The study also presented findings on a researcher-designed signaling test that did not meet WWC group design standards. 
The test instructed students to supply omitted signaling words within a short comparative text. Because instruction on signaling 
words was provided to students in the /TSS intervention group but not to students in the comparison group, and students were 
assessed on their use of appropriate signaling words, the WWC determined this test to be overaligned with the intervention. 


The study reported findings for a subsample of fourth- and fifth-grade classrooms that implemented /TSS with high fidelity, 
as defined by the authors. Three criteria were used to identify high-fidelity implementation: (1) teachers contacted the 
research team to follow up on the progress of their students; (2) teachers were able to monitor and answer questions from 
students, as defined by observers; and (3) students used /7SS for at least 30 minutes per week for at least 6 months. These 
findings are not presented in this report because high-fidelity implementation is not an eligible subgroup of interest under 
the Adolescent Literacy protocol (version 4.0). 


The study authors also report non-academic outcomes for the fifth-grade sample, including computer attitudes, learning 
self-efficacy, reading self-efficacy, and structure strategy self-efficacy. These findings were not eligible for review under the 
Adolescent Literacy protocol (version 4.0). 


Additional The research team conducted /TSS training sessions for the teachers of intervention classrooms during the 2009-10 
implementation academic year. Teachers in comparison classrooms were offered the same professional development after the study 
details was completed. 


Research details for Wijekumar et al. (2017) Findings from Wijekumar et al. (2017) show evidence of a sta- 
Wijekumar, K., Meyer, B..F., & Lei, P. (2017). Web-based text tistically significant positive effect of ITSS on comprehension 
: fo a : (Table 6). These findings are based on eight outcomes and 
structure strategy instruction improves seventh graders (Rosendentei® The Radius and neacarcideiaaks / 
content area reading comprehension. Journal of Educa- 3 ; eA ee 


tional Psychology, 109(6), 741-760. Retrieved from https:// oan aie eee pe renee oe 
eric.ed.gov/?id-E]1149967. primary study listed above. See the References section, 


which begins on page 11, for a list of all related publications. 


Table 6. Summary of findings from Wijekumar et al. (2017) 


Meets WWC group design standards with reservations 


Study findings 
; Sample Average Improvement Statistically 
Outcome domain size effect size index significant 
Comprehension 1,868 students 0.23 +9 Yes 


Table 7. Description of study characteristics for Wijekumar et al. (2017) 


WWC evidence 
rating 


Setting 


Methods 


Study sample 


Intervention 


group 


Comparison 
group 


Outcomes and 
measurement 


Meets WWC Group Design Standards With Reservations. This is a cluster randomized controlled trial with compromised 
random assignment that satisfies the baseline equivalence requirement for the individuals in the analytic intervention and 
comparison groups. 


The study included seventh-grade classrooms from 25 rural and suburban middle schools across two states in the 
United States. 


The study used a cluster randomized controlled trial design. A volunteer sample of 108 seventh-grade classrooms from 25 
schools were randomly assigned to either provide supplemental instruction using /TSS (59 classrooms with 1,415 students) 
or to implement business-as-usual classroom instruction in the study year and implement /TSS the following academic 
year (49 classrooms with 1,074 students). Altogether, 2,489 students from 25 schools were subject to random assignment. 
Random assignment occurred separately within each school. If a school had an odd number of classrooms, it was matched 
to a similar school with an odd number of classrooms and the random assignment then took place across these schools. All 
students within study classrooms were then asked to provide consent to participate in the study. The random assignment 
was compromised because students were excluded from the analytic sample if they used /TSS for less than 30 minutes 
during the school year. 


The number of students in the analytic sample differs by outcome, ranging from 1,716 to 1,868 students. 


Across the 25 study schools, 42% of the student population was eligible for free or reduced-price lunch and 8% were racial/ 
ethnic minorities. The analytic sample was about 48% female, and 53% were from rural school districts. The remainder 
were from suburban school districts. 


Students in the intervention group received the /TSS program over the course of the 2010-11 school year. /TSS was used in 
intervention classrooms for 30 to 45 minutes a week over 6 to 7 months, which was lower than the developer-recommended 
dosage, as a partial substitute for the regular language arts curriculum. 


Students in comparison classrooms received the typical language arts curriculum, which was the same curriculum used by 
the intervention group classrooms within the same school except for the partial substitution of /TSS. Total daily and weekly 
amounts of language arts instruction were the same for both intervention and comparison classrooms. 


Study authors reported findings on eight outcome measures that are eligible for review under the Adolescent Literacy topic 
area. All assessments were administered to seventh-grade students. For all measures, pretests were completed at the 
beginning of the school year and posttests were completed immediately after completing the study end of the intervention 
implementation (in April or May 2011). 


All outcomes were reviewed in the comprehension domain, and included the GSRT (Wiederholt & Blalock, 2000), and 
seven researcher-designed measures. For the researcher-designed measures described below, interrater agreement was 
within the acceptable threshold for the review and ranged from 86% to 100%: 


* Long Comparison Text: Competence Test. This assessment measured whether students recalled the problem, its 
cause, and its solution described in the original text. Competency scores ranged from 1 to 8, and described a student's 
use of the structure strategy to organize correct ideas. The long comparison text used in this assessment contained 
527 words, 33 sentences, and 134 idea units. 


¢ Long Comparison Text: Number of Issues Test. This assessment counted the number of issues that students 
correctly contrasted between two objects described in the passage presented in the Long Comparison Text: 
Competence Test. This assessment was scored on a 9-point scale. 


* Short Comparison Text: Competence Test. This assessment was also used in Wijekumar et al. (2012). It tested 
the student's ability to recall the correct problem, its cause, and its solution. Both pretest and posttest comparison 
passages used in this assessment had 128 words, 15 sentences, and 98 idea units. 


* Short Comparison Text: Number of Issues Test. This assessment counted the number of issues students correctly 
contrasted between two objects described in the passage presented in the Short Comparison Text: Competence Test. 
This assessment was rated on a 6-point scale. 


* Short Comparison Text: Main Idea Competence Test. This assessment was also used in Wijekumar et al. (2012); it 
was called the Short Comparison Text: Main Idea Test in the previous study. The assessment asked students to write a 
two-sentence main idea after reading a passage from the Short Comparison Text: Competence Test. The assessment 
measured students' competence using the comparison structure and their ability to organize ideas in the passage. It 
was rated on a 6-point scale. 


Outcomes and 
measurement 
(continued) 


Additional 
implementation 
details 


¢ Short Comparison Text: Main Idea Test, Number of Issues Test. This assessment asked students to write a 
two-sentence main idea after reading a passage from the Short Comparison Text: Competence Test. The 
assessment measured the number of issues compared. 


¢ Problem/Solution Text: Competence Test. This was the same assessment used in Wijekumar et al. (2012) and 
measured whether students recalled the problem, its cause, and its solution. The problem/solution text had two 
passages that each had 98 words and 72 idea units. 


Wijekumar et al. (2017) presented findings on the same researcher-designed signaling test used in Wijekumar et al. (2012). 


Again, the WWC determined this test to be overaligned for the same reason. Four assessments that required students to 
identify the top-level structure (such as problem and solution) were also determined to be overaligned because students 
received higher scores if they used signaling words in their answers. Because these outcomes might not provide an 
accurate indication of the effect of TSS, they do not meet WWC requirements. 


Teachers in the intervention group received 3 hours of training at the beginning of the academic year. Teacher aides were 
hired at each school to assist with /TSS implementation at the computer lab. 
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Endnotes 


'The descriptive information for this intervention comes 
from Wijekumar et al. (2012, 2017). The What Works 
Clearinghouse (WWC) requests developers to review the 
intervention description sections for accuracy from their 
perspective. The WWC provided the developer with the 
intervention description in March 2019, and the WWC 
incorporated feedback from the developer. Further 
verification of the accuracy of the descriptive information 


for this intervention is beyond the scope of this review. 


? At the time of this report writing, two studies were 
available on the effectiveness of the web-based version of 
ITSS, which is the focus of this report. Research on other 
versions of the intervention teaching structure strategy, 
such as those using human tutors, has a longer history. 
Because these versions of the intervention might differ 
in effectiveness from the web-based version of ITSS, this 
literature did not contribute to the findings in this report. 


3This report was revised in April 2021 to incorporate 
feedback on the definitions of outcome measures provided 
by the authors of Wijekumar et al. (2017). These revisions 
did not result in any changes to the data and conclusions 
presented in the original report. 


‘The literature search reflects documents publicly available 
by March 2019. Reviews of the studies in this report used 
the standards from the WWC Procedures and Standards 
Handbook (version 4.0) and the Adolescent Literacy review 
protocol (version 4.0). The evidence presented in this report 
is based on available research. Findings and conclusions 
could change as new research becomes available. 


5 The effects of ITSS are not known for other outcome 
domains within the Adolescent Literacy topic area, 
including alphabetics, reading fluency, general literacy 
achievement, writing conventions, writing productivity, 
and writing quality. 

6The WWC conducted a single study review of Wijekumar 
et al. (2012) in July 2013. This study presented findings 
on fourth-grade students only. For the present review, 
the WWC combined ITSS findings from fourth- and fifth- 
grade students reported in Wijekumar et al. (2012, 2014), 
respectively, because they met WWC group design criteria 
for being part of the same study. Specifically, findings 
reported for fourth- and fifth-grade students shared 
the same group formation procedures, the same data 
collection and analysis procedures, and the same research 
team. For more information, please see page 9 of the WWC 
Procedures Handbook (version 4.0). 


7The combined findings for the fourth- and fifth-grade 
students (Wijekumar et al., 2012, 2014) meet WWC group 
design standards without reservations because they are 
based on a randomized controlled trial with low attrition. 
Separate findings for fourth-grade students (Wijekumar 
et al., 2012) and fifth-grade students (Wijekumar et al., 
2014) also meet WWC group design standards without 
reservations. Although the authors reported the findings 
separately for each grade, the WWC combined findings 
from both grades because they are based on the same 
study as defined by the WWC. The WWC reviewed findings 
for each grade level as supplementary findings that are 
available on the WWC website (https://whatworks.ed.gov). 


8’ The number of fifth-grade students in the study at the 
time of random assignment is unknown. Authors reported 
that 98% of the students subject to random assignment 
provided consent to participate, so even under the most 
conservative assumptions, this subgroup has low attrition 
by WWC standards. 


® Wiederholt, J. L., & Blalock, G. (2000). GSRT: Gray Silent 
Reading Tests. Austin, TX: Pro-Ed. 


il 


0The WWC had conducted a previous review of Wijekumar 
et al. (2017), as this was a Department of Education-funded 
evaluation. This previous review resulted in the same 
rating as the present review, although there were some 
differences in which findings were reported as primary and 
supplementary due to differences in the review protocols 
used to guide these reviews. 
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